Random Vectors

Random vectors

Definition 1 (Projection). For a vector \(x \in \R^n\), we can define \(\pi_i: \R^n \to \R\) is the projection of an \(n\)-length vector onto its \(i\)-th component, such that \(\pi_i(x) = x_i\).

Definition 2. The Borel sigma algebra over space \(\R^n\) is defined as the smallest sigma algebra generated by the family \((\pi_i^{-1}(B_x): x \in \R, i \in [n])\) and is denoted by \(\sB(\R^n)\). The elements of the Borel sigma algebra are called Borel sets.

Remark 1. By definition of \(\sB(\R^n)\), projection \(\pi_i: \R^n \to \R\) is a Borel measurable function for all \(i \in [n]\). For a subset \(A \subseteq \R\) and projection \(\pi_i: \R^n \to \R\), we can write Thus, for any \(A \in \sB(\R)\), we have \(\pi_i^{-1}(A) \in \sB(\R^n)\).

Definition 3 (Random vectors). Consider a probability space \((\Omega, \sF, P)\) and a finite \(n \in \N\). A random vector \(X: \Omega \to \R^n\) is an \(\sF\)-measurable mapping from the sample space to an \(n\)-length real-valued vector. That is, for any \(x \in \R^n\), we have

Example 4 (Tuple of indicators). Consider a probability space \((\Omega, \sF, P)\), a finite \(n \in \N\), and events \(A_1, \dots, A_n \in \sF\). We define a mapping \(X: \Omega \to \set{0,1}^n\) by \(X_i(\omega) \triangleq \Ind{A_i}(\omega)\) for all outcomes \(\omega \in \Omega\). Let \(x \in \R^n\), then we can write \(A_X(x) = \cap_{i=1}^n\Ind{A_i}^{-1}(-\infty, x_i]\). Recall that It follows that the inverse image \(A_X(x)\) lies in \(\sF\), and hence \(X\) is an \(\sF\)-measurable random vector.

Theorem 5. Consider a probability space \((\Omega, \sF, P)\), and a finite \(n \in \N\). A mapping \(X: \Omega \to \R^n\) is a random vector if and only if \(X_i \triangleq \pi_i \circ X: \Omega \to \R\) are random variables for all \(i \in [n]\).

Proof. Proof. We will first show that \(X: \Omega \to \R^n\) implies that \(\pi_i \circ X\) is a random variable for any \(i \in [n]\). For any \(i \in [n]\) and \(x_i \in \R\), we take \(x = (\infty, \dots, x_i, \dots, \infty)\). This implies that \(\pi_i^{-1}(-\infty, x_i] = \R \times \dots \times (-\infty, x_i] \times \dots \times \R \in \sB(\R^n).\) Further, defining \(A_{X_i}(x_i) \triangleq X_i^{-1}(-\infty, x_i]\), we observe from the definition of random vectors that

We will next show that if \(X_i: \Omega \to \R\) is a random variable for all \(i \in [n]\), then \(X \triangleq (X_1, \dots, X_n): \Omega \to \R^n\) is a random vector. For any \(x \in \R^n\), we have \(A_{X_i}(x_i) = X_i^{-1}(-\infty,x_i] \in \sF\) for all \(i \in [n]\), from the definition of random variables. From the closure of event set under countable intersections, we have ◻

Distribution of random vectors

Definition 6. Consider a probability space \((\Omega, \sF, P)\) and a finite \(n \in \N\). The joint distribution function of a random vector \(X: \Omega \to \R^n\) is defined as the mapping \(F_X: \R^n \to [0,1]\) such that

Example 7 (Tuple of indicators). Consider a probability space \((\Omega, \sF, P)\), a finite \(n \in \N\), and events \(A_1, \dots, A_n \in \sF\), that define the random vector \(X \triangleq (\Ind{A_1}, \dots, \Ind{A_n})\). For any \(x \in \R^n\), we can define index sets \(I_0(x) \triangleq \set{i \in [n]: x_i < 0}\) and \(I_1(x) \triangleq \set{i \in [n]: x_i \in [0,1)}\), and write the joint distribution function for this random vector \(X\) as

Definition 8. For a random vector \(X: \Omega \to \R^n\) defined on the probability space \((\Omega, \sF, P)\) and \(i \in [n]\), the distribution of the \(i\)th random variable \(X_i \triangleq \pi_i\circ X: \Omega \to \R\) is called the \(i\)th marginal distribution, and denoted by \(F_{X_i}: \Omega \to [0,1]\).

Example 9 (Tuple of indicators). Consider a probability space \((\Omega, \sF, P)\), a finite \(n \in \N\), and events \(A_1, \dots, A_n \in \sF\), that define the random vector \(X \triangleq (\Ind{A_1}, \dots, \Ind{A_n})\). The \(i\)th marginal distribution is given by

Corollary 10 (Marginal distribution). Consider a random vector \(X: \Omega \to \R^n\) defined on a probability space \((\Omega, \sF, P)\) with the joint distribution \(F_X: \R^n \to [0,1]\). The \(i\)th marginal distribution and can be obtained from the joint distribution of \(X\) as

Proof. Proof. For any \(i \in [n]\) and \(x_i \in \R\), we have \(X_i^{-1}(-\infty, x_i] = A_X(x)\) for \(x = (\infty, \dots, x_i, \dots, \infty)\) from [eqn:RandomVector:MarginalEvent]. ◻

Lemma 11 (Properties of the joint distribution function). Consider a random vector \(X: \Omega \to \R^n\) defined on the probability space \((\Omega,\sF,P)\). The associated joint distribution function \(F_X: \R^n \to [0,1]\) satisfies the following properties.

1. For \(x, y \in \R^n\) such that \(x_i \le y_i\) for each \(i \in [n]\), we have \(F_X(x) \le F_X(y)\).

2. The function \(F_X(x)\) is right continuous at all points \(x \in \R^n\).

3. The lower limit is \(\lim_{x_i \to -\infty}F_X(x) = 0\), and the upper limit is \(\lim_{x_i \to \infty, i \in [n]}F_X(x) = 1\).

Proof. Proof. Consider a random vector \(X: \Omega \to \R^n\) defined on the probability space \((\Omega, \sF, P)\) and any \(x \in \R^n\).

1. We can verify that \(A_X(x) = \cap_{i=1}^nA_{X_i}(x_i) \subseteq \cap_{i=1}^nA_{X_i}(y_i) = A(y)\). The result follows from the monotonicity of probability measure.

2. The proof is similar to the proof for single random variable.

3. The event \(A_X(x) = \emptyset\) when \(x_i = -\infty\) for some \(i \in [n]\) and \(A_X(x) = \Omega\) when \(x_i = \infty\) for all \(i \in [n]\), hence the result follow.

 ◻

Example 12 (Probability of rectangular events). Consider a probability space \((\Omega, \sF, P)\) and a random vector \(X: \Omega \to \R^2\). Consider the points \(x \le y\in \R^2\) and the events The marginal probabilities are given by Writing \(x = (x_1,x_2)\) and \(y = (y_1,y_2)\), we observe that the end points of the rectangular event \(B_1\cap B_2\) are points \(x, (y_1,x_2), y, (x_1, y_2)\). Therefore, we can write this event as Hence, we can write the probability of this rectangular event as

Event space generated by random vectors

Definition 13. Consider a probability space \((\Omega, \sF, P)\) and a finite \(n \in \N\). The event space generated by a random vector \(X:\Omega \to \R^n\) is the smallest \(\sigma\)-algebra generated by the collection of events \((A_X(x): x \in \R^n)\) and denoted by \(\sigma(X) \triangleq \sigma(A_X(x):x \in \R^n)\).

Theorem 14. Consider a probability space \((\Omega, \sF, P)\), a finite \(n \in \N\), a random vector \(X: \Omega \to \R^n\), and its projections \(X_i \triangleq \pi_i \circ X\) for all \(i \in [n]\). Then, \(\sigma(X) = \sigma(X_1, \dots, X_n)\).

Proof. Proof. Recall that \(\sigma(X)\) is generated by the family \((A_X(x): x \in \R^n)\) and \(\sigma(X_1, \dots, X_n)\) is generated by the family \((A_{X_i}(x_i): x_i \in \R, i \in [n])\). We first show that \(A_{X_i}(x_i) = A_X(x)\) for \(x = (\infty, \dots, x_i, \dots, \infty)\), and hence \(\sigma(X_1, \dots, X_n) \subseteq \sigma(X)\). We then show that \(A_X(x) = \cap_{i=1}^nA_{X_i}(x_i)\), and hence \(\sigma(X) \subseteq \sigma(X_1, \dots, X_n)\). ◻

Example 15 (Tuple of indicators). Consider a probability space \((\Omega, \sF, P)\), a finite \(n \in \N\), and events \(A_1, \dots, A_n \in \sF\), that define the random vector \(X \triangleq (\Ind{A_1}, \dots, \Ind{A_n})\). The \(\sigma(X) = \sigma(A_i: i \in [n])\).

Independence of random variables

Definition 16. A family of collections of events \((\sA_i \subseteq \sF: i \in I)\) is called independent, if for any finite set \(F \subseteq I\) and \(A_i \in \sA_i\) for all \(i \in F\), we have

Definition 17 (Independent and identically distributed). A random vector \(X: \Omega \to \R^n\) defined on the probability space \((\Omega, \sF, P)\) is called independent if The random vector \(X\) is called identically distributed if each of its components have the identical marginal distribution, i.e.

Remark 2. Independence of a random vector implies that events \((A_{X_i}(x_i): i \in [n])\) are independent for any \(x \in \R^n\). Defining families \(\sA_i \triangleq (A_{X_i}(x): x \in \R)\) for all \(i \in [n]\), we observe that the families \((\sA_1, \dots, \sA_n)\) are mutually independent.

Remark 3. In general, if two collection of events are mutually independent, then the event space generated by them are independent. This can be proved using Dynkin’s \(\pi\)-\(\lambda\) Theorem.

Theorem 18. For an independent random vector \(X: \Omega \to \R^n\) defined on a probability space \((\Omega, \sF, P)\), the event spaces generated by its components \((\sigma(X_i): i \in [n])\) are independent.

Proof. Proof. For an we define a family of events \(\sA_i \triangleq (X_i^{-1}(-\infty, x]: x\in \R)\) for each \(i \in [n]\). From the definition of independence of random vectors, the families \((\sA_i \subseteq \sF: i \in [n])\) are mutually independent. Since \(\sigma(\sA_i) = \sigma(X_i)\), the result follows from the previous remark. ◻

Definition 19 (Independent random vectors). To random vectors \(X:\Omega \to \R^n\) and \(Y: \Omega\to\R^m\) defined on the same probability space \((\Omega, \sF, P)\) are independent, if the collection of events \((A_X(x): x \in \R^n)\) and \((A_Y(y): y \in \R^m)\) are independent, where \(A_X(x) \triangleq \cap_{i=1}^nX_i^{-1}(-\infty, x_i]\) and \(A_Y(y) \triangleq \cap_{i=1}^mY_i^{-1}(-\infty, y_i]\).

Example 20 (Independent random vectors). Consider a set of vectors \(\sX = \set{(0,0,1),(1,0,0)}\subseteq \R^3\). Consider two independent coin tosses, such that \(\Omega = \set{H,T}^2, \sF = \cP(\Omega)\) and \(P(\omega) = p^{k_2(\omega)}(1-p)^{2-k_2(\omega)}\), where \(k_2(\omega) = \sum_{i=1}^2\SetIn{\omega_i = H}\). We define random vectors

2 &X = (0,0,1) + (1,0,0),& &Y = (0,0,1) + (1,0,0).

We can verify that \(X,Y: \Omega \to \R^3\) are mutually independent random vectors, though we can also check that \(X_1, X_3\) are dependent random variables and so are \(Y_1, Y_3\).

Discrete random vectors

Definition 21 (Discrete random vectors). If a random vector \(X: \Omega \to \sX_1 \times \dots \times \sX_n \subseteq \R^n\) takes countable values in \(\R^n\), then it is called a discrete random vector. That is, the range of random vector \(X\) is countable, and the random vector is completely specified by the probability mass function

Remark 4. For an independent discrete random vector \(X: \Omega \to \R^n\), we have \(P_X(x) = \prod_{i=1}^nP_{X_i}(x_i)\) for each \(x \in \R^n\).

Example 22 (Multiple coin tosses). For a probability space \((\Omega, \sF, P)\), such that \(\Omega = \set{H,T}^n, \sF = 2^\Omega, P(\omega) = \frac{1}{2^n}\) for all \(\omega \in \Omega\).

Consider the random vector \(X: \Omega \to \R\) such that \(X_i(\omega) = \SetIn{\omega_i = H}\) for each \(i \in [n]\). Observe that \(X\) is a bijection from the sample space to the set \(\set{0,1}^n\). In particular, \(X\) is a discrete random variable.

For any \(x \in [0,1]^n\), we can write \(N(x) = \sum_{i=1}^n\Ind{[0,1)}(x_i)\). Further, we can write the joint distribution as We can derive the marginal distribution for \(i\)-th component as Therefore, it follows that \(X\) is an random vector.

Continuous random vectors

Definition 23 (Joint density function). For jointly continuous random vector \(X: \Omega \to \R^n\) with joint distribution function \(F_X: \R^n \to [0,1]\), there exists a joint density function \(f_X: \R^n \to [0,\infty)\) such that \(f_X(x) = \frac{d^n}{dx_1\dots dx_n}F_X(x)\), and

Remark 5. For an independent continuous random vector \(X: \Omega \to \R^n\), we have \(f_X(x) = \prod_{i=1}^nf_{X_i}(x_i)\) for all \(x \in \R^n\).

Example 24 (Gaussian random vectors). For a probability space \((\Omega, \sF, P)\), Gaussian random vector is a continuous random vector \(X: \Omega \to \R^n\) defined by its density function where the mean vector \(\mu \in \R^n\) and the positive definite covariance matrix \(\Sigma \in \R^{n \times n}\). The components of the Gaussian random vector are Gaussian random variables, which are independent when \(\Sigma\) is diagonal matrix and are identically distributed when \(\Sigma = \sigma^2 I\).